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1.   INTRODUCTION 

The  following  document  describes  various  aspects  of  the 
ACSES  quiz  system  as  it  is  currently  implemented  on  the  PLATO  CAI 
system.  With  the  intent  of  an  introductory  manual,  the  content  herein 
is  directed  primarily  to  the  potential  quiz  system  user.  Appendices 
conclude  the  manual  by  presenting  selected  system  specifics  to  allow 
a  user  a  more  comprehensive  view  of  how  the  system  works.  A  close 
investigation  of  this  paper  should  thus  yield  sufficient  details  of 
the  quiz  system's  operation  to  enable  knowledgeable  utilization  of 
that  which  the  system  offers. 


2.   SYSTEM  PURPOSE  AND  FUNCTION 

The  quiz  system  was  designed  and  implemented  by  this  author 
to  fulfill  two  purposes  as  proposed  by  R.  G.  Montanelli,  Jr. :  1)  to 
provide  a  student  taking  a  PLATO  computer  science  lesson  with  both  the 
means  to  assess  how  well  he  or  she  learned  the  material  that  the  lesson 
is  intended  to  cover  and  a  tool  to  utilize  for  aid  in  learning  the  topic 
at  hand;  2)  to  provide  members  of  the  Department  of  Computer  Science 
staff  with  a  means  to  assess  how  effective  and  how  thorough  a  PLATO 
computer  science  lesson  is  at  teaching  its  topic. 


3.   FULFILLMENT  OF  SYSTEM  PURPOSE 

To  fulfill  this  dual  purpose,  a  pool  of  PLATO  quizzes 
(see  Appendix  D)  has  been  assembled  for  administration  to  students  at 
the  conclusions  of  appropriate  computer  science  lessons.  Each  quiz 
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has  been  designed  to  pertain  to  some  well-defined  topic  within  the 
computer  science  field,  and  each  quiz  question  has  been  selected  by 
the  quiz  author  to  test  pertinent  details  of  the  topic's  content. 
Incorporated  into  each  quiz  is  a  data  collection  facility  to  record 
question  responses  as  made  by  students  during  their  initial  attempts 
at  the  quiz. 

Access  to  the  pool  of  quizzes  is  provided  via  the  quiz 
system  monitor  to  authorized  individuals  employing  PLATO  computer 
science  lessons  for  instructional  purposes.   Such  a  user  can  select 
a  quiz  of  the  desired  content  area,  interactively  design  the  quiz  to 
best  suit  his  or  her  chosen  lesson's  needs,  view  the  quiz  exactly  as  a 
student  will,  and  finally,  via  only  a  few  keypresses,  actually  "attach" 
the  quiz  to  the  chosen  instructional  lesson.   The  latter  task  does, 
though,  require  a  bit  more  effort  as  changes  must  be  made  to  the  code 
of  the  instructional  lesson  to  enable  an  interface  with  the  quiz. 
However,  these  changes  are  only  minor  and  are  clearly  outlined  to  the 
user  when  quiz  attachment  is  arranged. 

A  look  at  Diagram  1  should  be  adequate  to  explain  how  the 
instructional  lesson/quiz  interface  operates.  At  the  time  a  quiz  is 
to  be  administered  to  a  student,  control  is  transferred  from  the 
instructional  lesson  to  a  quiz  system  program  that  functions  as  a  link 
to  the  quiz  (arrow  A).   This  program  determines  which  quiz  of  those 
available  in  the  system  is  to  be  presented  for  this  particular  instructional 
lesson,  and  control  is  transferred  to  it  (arrow  B).   Interaction  between 
the  system  program  and  the  lesson  that  produces  the  quiz  occurs  at  various 
points  during  quiz  administration  (arrow  C).   Following  a  review  of  the 
corrected  quiz,  a  student  is  returned  to  the  system  program  a  final 
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time  (arrow  D)  where  return  to  the  appropriate  instructional  lesson 
is  provided  (arrow  E). 

After  the  few  minor  changes  are  made  to  the  instructional 
lesson  and  a  quiz  has  been  attached  via  the  system  monitor,  quiz 
administration  is  ready  to  commence.   Each  student  from  then  on  will 
have  unrestricted  access  to  the  quiz  and  will  be  informed  of  the  average 
score  obtained  on  the  quiz  by  other  members  of  his  or  her  course.   Each 
system  user  from  then  on  will  have  unrestricted  access  (via  the  quiz 
system  monitor)  to  data  that  is  accumulated  for  each  quiz  question  and 
will  be  informed  (for  all  courses  that  have  used  the  computer  science 
lesson)  of  both  the  average  quiz  scores  and  the  average  amount  of  time 
needed  for  taking  the  quiz.   And  each  authorized  member  of  the  Department 
of  Computer  Science  staff  from  then  on  will  have  unrestricted  access  also 
to  the  data  accumulated  for  each  quiz  question  to  facilitate  analysis  of 
the  effectiveness  and  completeness  of  both  the  instructional  lesson  and 
the  quiz. 

At  the  conclusion  of  course  instruction,  utilized  quizzes 
should  be  detached  from  instructional  lessons  to  clear  accumulated 
course  data.   This  procedure,  which  can  be  accomplished  through  the 
quiz  system  monitor,  leaves  all  such  data  irretrievable.   Thus 
important  information  should  be  recorded  prior  to  the  fatal  keypresses. 
During  the  time  that  a  quiz  is  detached  from  an  instructional  lesson, 
no  reversal  of  the  quiz-accomodating-alterations  that  were  made  earlier 
to  the  lesson  are  necessary.  Transfer  of  control  to  the  quiz  system 
program  so  to  administer  the  quiz  (arrow  A  of  Diagram  1)  will  simply 
result  in  the  display  of  the  message:   "No  quiz  currently  exists  for 
this  lesson".   Control  is  then  immediately  returned  from  whence  it  was 
yielded  (arrow  E  of  Diagram  l). 

-h- 


k.      SCHEME  OF  QUIZ  ADMINISTRATION 

All  quizzes  are  designed  to  be  of  similar  structure  (see 
Appendix  A),  thus  some  aspects  of  the  interaction  that  occurs  between 
a  student  and  a  quiz  are  identical  in  all  cases.  The  common  sequence 
involved  is  briefly  sketched  below  (examples  are  taken  from  a  quiz  on 
FORTRAN  character  manipulation): 

1)  once  the  quiz  system  program  verifies  the  existence  of 
a  quiz  for  a  particular  instructional  lesson,  a  page 
detailing  the  quiz's  purpose  is  presented  to  the  student 
(Display  l) 

2)  quiz  questions  are  then  administered;  questions  may  be 
skipped  if  the  student  so  desires,  and  all  questions 
may  be  reanswered  in  case  an  error  was  made  (Display  2) 

3)  when  the  student  decides  that  his  or  her  attempt  at  the 
quiz  is  complete,  he  or  she  may  advance  to  the  presentation 
of  the  corrected  quiz  which  is  accompanied  by  clarifying 
explanations  (Display  3) 

k)     lastly  the  student  is  informed  of  his  or  her  final  quiz 
score  as  well  as  the  average  score  received  by  others  in 
his  or  her  course  (Display  k) 


5.   HOW  TO  USE  THE  QUIZ  SYSTEM  MONITOR 

Entrance  into  the  quiz  system  monitor  is  achieved  via 
execution  of  program  "csqdata".  An  individual  must  receive  authorization 
from  the  quiz  system  administrator  (see  section  7)  before  use  of  the 
system's  powers  is  permitted  (Display  5). 
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Dear  anderson, 

You  a.rve  .about  to  take  a  quiz  over  the  material 
in  lesson  fort char.   This  quiz,  which  was  not  prepare 
by  the  author  (s)  of  fort char,  will  NOT  count  a a  part 
of  your  grade.   It  lias  instead  the  following  two 
pi  i r poses: 

1)  to  give  you,  the  student,  somewhat  of  an  idea 
of  how  well  you.  learned  the  material  that  the 
lesson  is  intended  to  cover.   When  you.  complete 
the  quiz,  you'  ]  1  be  shown  how  wel  1,  on  "che 

average,  previous  students  have  clone i 


the  staff 
^  and  tho 


=  ,-,rri^i„ha-l- 


Rnoui  i  edge 


is     dt::'.  1  gT  ied      tO     t€ct     wl  Id.  i       t  f  l<r      1  €itiOn 

ed  to  cover.   H  low  score  ma1-1  indicat 

the  lesson  rather  than  e    lac):  of 
>n  y<  'U.r  part . 


NOTE:  The  correct  answers  to  each  question  will 
not  be  revealed  until  you  complete  the  quiz. 

Press:  -NEXT-  to  begin  the  quiz 

-BftCK-  to  postpone  the  quiz  until  later 


Display  1:   PAGE  PRESENTED  EACH  STUDENT  PRIOR  TO  QUIZ 


-6- 


Quest  ion    3    o f    9 


i" -■rtchar    qui 


Lpaili^ 


Fo  1  1  ow  i  ng   execut  i  on   o  f    t he    f  o  ]  1  ouj  i  n;; 

RERL   D/'HIV 

CHARACTER  G ,  H#  9 ,  B  *  7  /  '  Chi  I CRGO 

INTEGER  A/  'ILL. INI  V 

G=B 

H=B 


triable  G  c< 

1)  CHIC 

2)  CHIC 


f.  M'-jf 


ntaini 
RGO 


;tl 


n<  -n* 


,--■,  +' 


Jenotes  answer  you've  given 


b  denotes  a  blank  space. 


Press:  -SHIFT  DRTR-  to  have  qui 
-BACK-  for  question  2 
-NEXT-  for  question  4 


z  graded 


Display  2:   SAMPLE  QUIZ  QUESTION 
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I    Quest  i  on  3  <  •  i. 


1  :>rtchar  qu  1  z 


nts 


Fo  i ; 

:>wing  execution  of  the  following  code-- 

S i nee  t  he 

length  o 

acter  var 

f  char- 
lable  G 

CHRRRCTER  G,     B* 7/' CHICAGO', 

i s  not  exp I i c i t 1 

>-•  stated,  i 

t  defaults  to  one  byte.   So  when  7-1 

G=B 

character- long  B  is  assigned  to  G,j 

j  the  right -most  6  characters  3  re] 

|  truncated.; 

v*r  i  ab  1  e  G  cont a.  i  ns  exact  1  y : 


1  ] 


2 1  i. 


HICAGG 
HIC 


H'.-,U 


a  b 


*    denotes   answer   you've -given 
X    indicates   correct    answer 
b   denotes   a   blank    space 


Press:     -SHIFT   DRTR-    to   exit 
-BACK-     for   question    2 
-NEXT-     for    question    4 


Display  3:      SAMPLE  CORRECTED  QUIZ  QUESTION 
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Total    score   received  on   quiz;     16/18 

Average   score   of   the    3    people   of   course 
cs  1 0  1  ws   who   have   taken   the   quiz:     9.333X18 


To   review   the   quiz    again,    press    -BftCK- . 
To   exit,    press    -NEXT-. 


Display  k:      SAMPLE  PAGE  PRESENTED  EACH  STUDENT 
AFTER  REVIEW  OF  CORRECTED   QUIZ 
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You  have  just  entered  the 


X 


1  2; 


-J  v 


m 


Authored   by   Rich   Finder  so 


SORRY 


You  are  not  allowed  access  to  this  lesson.   To  be 
considered  for  such,  state  who  you  3.re.   and  whv  you 
d e s  1  r e  a c c e s s  v  1  a  p - n o t e s  t c •  a n d er  son  of  c s a 


Copyright  ycj.     1-"'^5  ■:'•-->  -J  ■•■■*  t>-.  ,.=  + 


It  ■  1  -.  pf-c  1 


,  !   .   .  1   T1 


Tl 1 inoi 


Display  5:   TITLE  PAGE  OF  QUIZ  SYSTEM  MONITOR 

AS  IT  APPEARS  TO  UNAUTHORIZED  USERS 
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All  the  work  of  an  authorized  system  user  is  accomplished  via 
manipulation  of  five  options  (Display  6).  A  brief  description  of  each 
option  follows  below: 

Option  "l)  Attach  a  quiz  to  a  lesson" 

To  be  chosen  when  a  quiz  is  to  be  "attached"  to  an 
instructional  computer  science  lesson.  User  receives  access 
to  the  pool  of  available  quizzes  from  which  one  may  be  selected, 
designed,  and  "attached".   If  desired,  a  password  can  be 
specified  to  protect  the  quiz  from  detachment  by  other  system 
users.  A  complete  list  of  changes  that  must  be  made  to  the 
lesson  to  enable  proper  functioning  of  the  lesson/quiz  inter- 
face is  supplied. 

Option  "2)  Detach  a  quiz  from  a  lesson" 

To  be  chosen  when  a  quiz  is  to  be  "detached"  from  an 
instructional  computer  science  lesson.  User  must  supply 
appropriate  password  (if  one  was  specified  earlier  via 
option  l)  in  order  to  deal  the  lesson/quiz  interface  a  fatal 
blow. 

Option  "3)  View  list  of  lessons  with  quizzes" 

To  be  chosen  to  inspect  a  listing  of  all  computer  science 
lessons  currently  utilizing  a  system  quiz.   Lessons  with  a 
quiz  attached  that  for  some  reason  has  been  disabled  are 
identified. 

Option  "h)     View  a  quiz's  accumulated  data" 

To  be  chosen  to  inspect  student  data  accumulated  by  both 
the  quiz  and  the  quiz  system.  Available  data  includes  responses 
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Choor>e   what    you   want    t       i< 


1)  Attach   a   quiz    to   a    lesson 

2)  Detach   a   quiz    from   a    lesson 

3)  View    list    of    lessons   with   qui: 

4)  View   a   quiz's   a<  cumulated   dat- 

5)  Exit 


■  ]  \-\r    -  -    13.18.14.  DRTE     -  -    03X12  /  7  6 


Display  6:      OPTION  PAGE  OF  AUTHORIZED 
SYSTEM  MONITOR  USERS 
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made  by  students  to  each  quiz  question  (Display  7),  number 
of  students  of  each  course  that  have  taken  the  quiz,  average 
quiz  score  of  the  students  of  each  course,  and  the  average 
amount  of  time  required  by  students  of  a  course  to  complete 
the  quiz  (Display  8). 

Option  "5)  Exit" 

To  be  chosen  to  end  a  visit  to  the  quiz  system  monitor. 


6.   CURRENT  STATUS  OF  THE  QUIZ  SYSTEM  MONITOR 

Currently  only  quizzes  pertaining  to  selected  topics  of  the 
FORTRAN  programming  language  have  been  implemented  (see  Appendix  D). 
As  the  first  quizzes  of  the  system,  these  were  all  designed  and  developed 
from  objectives  used  to  develop  existing  FORTRAN  instructional  lessons; 
thus  very  little  versatility  was  provided  for  a  system  user  to  manipulate 
the  quiz  design  to  satisfy  other  lessons'  needs.  More  quizzes  are 
being  developed,  however,  and  those  quizzes  existing  are  continually 
being  improved.   The  availability  of  a  quiz  that  suits  a  user's  lesson's 
needs  should  therefore  be  investigated  via  entrance  into  the  quiz 
system  monitor. 


7.   QUIZ  SYSTEM  ADMINISTRATOR 

The  one  individual  who  has  complete  information  about  every- 
thing involving  the  quiz  system  is  the  quiz  system  administrator.   This 
person  (currently  this  author)  is  responsible  for  the  total  upkeep  of 
the  system  and  should  be  held  accountable  for  all  quiz  system  activities, 
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Quest  ion    3    o f 


fort  char    'quiz 


ntfi 


Today    is   08.  12/7 


Fo 1  I ow i ng   eyecut i on   of   t he    f o 1  1 ow l ng   cede - - 


RERL  D/'HIV 

CHRRRCTER  G , H#  9 , B*  7 /  'CHI CflGO  ' , 

INTEGER  R/' ILL INI  V 

G=B 

H=B 

variable  G  contains  exactly: 

■i          i,i  CHICAGO 

1    2)  CHIC 

i       i  hQi 

-. 

— - 

.  .    ,   . , 

i"  te 

numoer  at  tne  text  o!  each  possible  resf: 

onse; 

i  nd  i 

ca.tes  the  frequency  that  students  chose 

it . 

X    indicates   correct    answer 
b   denotes   a.   blank   space 


Press:     -SHIFT    DRTR-    to   exit 
-BACK-    for   question    2 
-NEXT-     for   que- t ion    4 


Display  7:      SAMPLE  DATA  ACCUMULATED  FOR 
A   QUIZ   QUESTION 
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fort subex  qu  1  z  dat a  -  0 6 / 15/76  t  hrough  0 8 / 1  2 / 7  6 

no .  who ' ve  ftve 
COURSE ta}cen_gu_iz  _s£2E^_ 

CS101WS                  13  11.69,16 

cslflljl                  15  11.47,16 

ge293                    1  6.00/16 

CS400  '                   3  6  9.39,16 

wdl06g                    6  7 . 3  3, 1 6 


* 


approximate   average    in   minutes 
Press    -NEXT-    to   choose   another    opt l or 


Display  8:      SAMPLE  ACCUMULATED  COURSE  DATA 
FOR  A  QUIZ 
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fiv 

e . 

1 1  me 

tak 

lDg_3J=iiz* 

6. 

6  2 

t  . 

20 

4. 

00 

36 

q 

8  3 

Specific  tasks  that  this  person  must  perform  are  reflected 
in  the  list  of  options  available  for  a  quiz  administrator's  work 
(Display  9)«   Options  1,  2,  k,   and  5  are  identical  in  function  to  the 
same-numbered  options  that  are  available  to  a  quiz  system  user;  the  other 
five  options  involve  tasks  exclusive  to  the  quiz  system  administrator 
to  ensure  smoothness  of  system  operation.   (see  Appendix  B) 

Any  problems  that  develop  or  any  questions  that  arise  concerning 
the  quiz  system  should  always  be  brought  to  the  attention  of  the  quiz 
system  administrator.   The  identity  of  this  indidivual  as  well  as 
information  detailing  how  he  or  she  may  be  reached  will  always  be 
specificed  on  the  title  page  of  the  quiz  system  monitor. 

8.   EVALUATION  OF  INSTRUCTIONAL  LESSONS 

The  potential  behind  the  analysis  of  accumulated  quiz  question 
data  for  purposes  of  locating  instructional  lesson  weaknesses  is  well- 
reflected  in  a  partial  examination  of  the  effects  of  the  initial  trial 
of  the  quiz  system  which  occurred  during  the  fall  semester  of  1975.   One 
quiz  which  was  prepared  to  test  selected  knowledge  of  character  manipulation 
in  the  FORTRAN  programming  language  was  presented  following  an  instructional 
lesson  of  the  same  topic.   The  instructional  lesson  had  been  used  in  two 
courses  by  approximately  800  students  prior  to  this  trial  and  had  under- 
gone various  revisions  based  on  information  obtained  during  the  courses. 
The  accumulation  of  question  responses  for  this  trial  was  restricted 
only  to  those  resi^onses  made  by  students  of  a  particular  introductory 
computer  science  course. 

Display  10  exemplifies  the  data  as  accumulated.   Since  all 
questions  were  designed  to  be  answered  correctly  by  all  students  if 
all  were  intelligently  attentive  while  taking  a  thorough  and  effective 

-16- 


Choose  what  you  want  to  do: 


1)  Attach  a  quiz  to  a  lesson 

2)  Detach  a  quiz  from  a  lesson 

3)  View  list  of  lessons  with  quizzes  <U 

4)  View  a  quiz's  accumulated  data 

5)  Exit 

6)  View   special    system   data 

7)  View    list    of   quiz    lessons    available 

8)  Add   a   new   quiz    lesson    (qg/g) 
r"'  Delete   a   quiz    lesson     (qg/g) 


attache?-  s 


TIME    --    13. 21.34, 


DATE"    --    08/12/76 


Display  9:      OPTION  PAGE  OF  QUIZ  SYSTEM 
ADMINISTRATOR 
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Qi  .est  i'  >n 


t"  Z     M.-ir 


qu  1  z 


T  .:■,  Jc 


12  19  ■ 


Following  execution  of  the  following 


u 


RERL   D/'HI ' ■ 
CHARACTER   G,hk  9,B* 
INTEGER   A/' ILL INI' 
G=B 
H=B 


CUK  RGC 


a  r  ]  ab  I  e  G  conta  i  ns  exact  1 
HICRGO 


"*  !   ■  •?- 


X  indicat' 


>rr 


answei 


b  denotes  a  blank  spa- 


P'r  e:r'-~  > 


SHIFT 

DATA-  to  exit 

BACK- 

for  quest i on 

NEXT- 

for  que? -t  ion 

Display  10:   RESPONSE  DATA  ACCUMULATED  DURING 
INITIAL  TRIAL  OF  SYSTEM 
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FORTRAN  character  manipulation  lesson,  some  problem  seemed  evident  due 
to  a  split  of  responses  among  three  of  the  five  alternatives.  Although 
eight  students  responded  correctly,  more  appeared  to  conclude  that  the 
length  of  variable  G  would  be  the  length  of  the  variable  assigned  to  it, 
and  many  could  find  no  satisfactory  choice  among  the  first  four  possibilities, 

Examination  of  the  instructional  lesson  revealed  two 
irregularities:  l)  nowhere  did  the  lesson  mention  that  the  default 
length  of  a  variable  declared  to  be  CHARACTER  in  type  is  one;  2)  a 
response  to  a  question  requiring  students  to  type  a  declaration  statement 
for  a  variable  that  could  accomodate  only  one  character  at  a  time  was  not 
judged  as  acceptable  unless  the  variable  length  was  explicitly  specified 
in  the  declaration.   Both  irregularities  clearly  represented  lesson 
deficiencies;  in  fact,  the  latter  clearly  represented  a  lesson  error. 

The  instructional  lesson  immediately  underwent  a  major  "face- 
lift"; the  above-mentioned  deficiencies  and  others  that  were  discovered 
through  similar  analyses  of  other  quiz  question  data*  dictated  a  complete 
lesson  restructuring.   Display  11  and  display  7  exemplify  data  as 
accumulated  during  spring  semester  1976  and  summer  session  1976, 
respectively,  following  the  instructional  lesson's  surgery.  The 
effects  of  the  changes  are  clearly  noticeable. 


* 
Other  deficiencies  discovered  included  no  use  of  special  characters 

and  no  discussion  of  the  use  of  character  strings  in  variables  declared 
other  than  CHARACTER. 
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9.   FINAL  NOTE 

Various  enhancements  to  and  improvements  of  the  quiz  system 
are  planned  for  the  future.   Such  alterations  that  are  foreseen  for 
the  "near"  future  should  be  of  a  scope  major  enough  to  invalidate  any 
information  presented  in  this  document.   All  occurrences  of  major 
system  changes  will  be  accompanied  by  explanatory  manual  amendments. 
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Quest  ior 


t'^-rtc.har    quiz 
Jay    is     5/2  s/ 76 


£_J^ii 


ni 


0<  let1 


Following  execution  of  the  following  code--- 

REAL  D/'HIV 

CHRRfiCTER  G , H* 9 , B*7/ ' CHICRGO ' 

INTEGER  R/' ILL INI'/ 

G  =  B 
H=B 


ar 1 ab 1 e  G  cent  a  1 ns  exact  1 


1 7 

1) 

(_ 

HICRGU 

j 

2.1 

r 

HIC 

.j 

-! ) 

■-- 

HUjU 

\ .-'     H  \ 

- 

j  K- 

C"  '•; 

n<  ne  0  i 

the  above 

?    r—> 

number 

-     3± 

the  left 

i"'     j-  _  _  1       -     —   '  [-   1 

r  ess:  r  ?~-e 

|  indi 

:1a  tes 

■   j"-;^ 

frequency 

>   that  students  cr 

_  .._..  __    -.4- 

nd 1 cates  correct  answer 
lenotes  a  blank  space 


Pr- 


-SHIFT   DftTfi-    to   exit 
-BRCK-    for   question    2 
-NEXT-     for    question    4 


Display  11:      RESPONSE  DATA  ACCUMULATED  FOLLOWING 
INSTRUCTIONAL  LESSON  ALTERATIONS 
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APPENDIX  A 

The  following  is  a  handout  as  prepared  by  this  author  for 
use  by  individuals  in  implementing  a  quiz-producing  lesson.   Its  content 
clearly  does  not  encompass  information  necessary  for  the  development  of 
a  "good"  criterion-referenced  quiz;  it  is  rather  intended  to  encompass 
the  needed  basics  behind  the  structure  of  such  a  quiz.   Since  the  hand- 
out was  intended  to  be  a  self-contained  document,  some  information 
presented  in  it  duplicates  information  as  presented  in  the  body  of 
this  manual. 

GUIDELINES  FOR  DEVELOPING  COMPUTER  SCIENCE  QUIZZES 

(Direct  questions  or  comments  to  Rich  Anderson  via  phone  or  send  a  p-note 
to  anderson  of  csa. ) 

1.   GENERAL  COMMENTS 

These  guidelines  exist  since  all  quizzes  written  for  use  with 
computer  science  instructional  lessons  are  tied  into  one  organizational 
system  that  monitors  and  controls  quiz  use.   Each  quiz  must  be  constructed 
to  conform  to  certain  requirements  which  will  allow  such  an  interface. 

Two  major  purposes  of  each  quiz: 

(1)  to  provide  a  student  with  both  a  means  to  assess  how 
well  he  or  she  learned  the  material  the  instructional 
lesson  is  intended  to  cover  and  a  tool  to  utilize  for 
help  at  learning  the  topic  at  hand. 

(2)  to  provide  members  of  the  Department  of  Computer  Science 
staff  with  a  means  to  assess  how  effective  and  how 
thorough  the  instructional  lesson  is  at  teaching 

the  topic 
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Each  quiz  should  be  designed  to  test  pertinent  aspects  of  some 
well-defined  topic  in  the  field  of  computer  science.  A  quiz  author  should 
avoid  viewing  an  instructional  lesson  on  the  topic  since  doing  so  may  bias 
the  choice  of  quiz  questions.   Contact  R.  I.  Anderson  for  direction  in 
determining  question  content. 

Quiz  questions  should  neither  be  "tricky"  nor  heavily  dependent 
upon  knowledge  of  other  materials  for  complete  student  understanding. 
A  well-formulated  quiz  question  should  be  responded  to  correctly  by  all 
students  if  each  was  intelligently  attentive  during  a  complete  and  effective 
instructional  lesson. 


2.   ORGANIZATION  OF  A  QUIZ 

Each  quiz -producing  lesson  must  perform  the  four  main  tasks 
which  are  discussed  below.   Each  task  must  begin  at  a  distinct  location 
in  the  lesson's  code  and  end  via  a  -jumpout-  to  the  quiz  system. 

Task  1  -  Administration  of  the  quiz  to  a  student 

Each  question  must  be  presented  to  a  student  to  allow  entry 
of  a  response.   If  a  student  has  previously  attempted  a  question  and 
decides  to  try  it  again,  his  or  her  original  answer  (if  one  had  been 
entered)  should  be  displayed  with  the  question,  and  a  new  response 
should  be  accepted  if  the  student  desires  to  enter  one. 

A  student  is  not  to  be  told  during  this  task  whether  or  not 
his  or  her  response  was  correct. 

When  a  student  decides  to  quit,  the  quiz  score  must  be  tallied 
and  if  this  is  the  first  time  this  student  has  taken  this  quiz,  student 
responses  must  be  stored.   Exit  is  then  made  via  a  -jumpout-  to  the 
system.  Following  this  exit,  a  student  can  decide  to  return  and  perhaps 
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change  more  answers;  provisions  must  thus  be  made  so  the  final  score 
and  stored  student  responses  reflect  any  possible  changes. 

Task  2  -  Review  of  the  corrected  quiz 

Each  question  must  in  some  manner  be  again  presented  to  a 
student  designating  both  the  correct  response  and  the  response  the 
student  entered  (if  indeed  the  student  did  respond).   The  student's 
scores  and  response  explanations  should  also  be  presented  if  deemed 
useful  to  a  student. 

New  answers  are  not  to  be  accepted  during  this  task. 

When  a  student  decides  to  quit,  exit  should  be  made  via 
a  -jumpout-  to  the  system.   Following  this  exit,  a  student  can  decide 
to  return  to  again  review  the  corrected  quiz. 

Task  3  -  Presentation  of  student  responses 

All  final  student  responses  should  have  been  retained  exactly 
(if  possible)  in  a  "permanent"  location  such  as  (and  probably)  lesson 
common.  Thus,  each  question's  responses  should  in  some  manner  be 
displayed  allowing  analysis  by  the  staff  to  locate  weaknesses  in  the 
instructional  lesson,  the  quiz,  and  the  students. 

Task  h  -  Accepting  quiz  specifications  from  system  user 
Comments  about  the  quiz  questions'  content  should  be  presented 
along  with  any  options  available  to  a  user  to  allow  him  or  her  some  say 
in  what  comprises  the  quiz.  A  keypress  should  be  all  that  is  necessary 
to:  take  a  system  user  into  the  quiz  as  a  student  would  view  it;  take  a 
system  user  into  reviewing  the  corrected  quiz  as  a  student  would  review 
it;  take  a  system  user  back  to  the  system  via  a  -jumpout-. 
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Much  of  the  content  of  the  pages  that  follow  is  derived  from 
a  handout  prepared  by  L.  R.  Whitlock  (1976).   The  basics  of  the  inter- 
face utilized  in  the  exam  system  to  enable  a  problem  generator /grader 
to  be  a  part  of  an  exam  have  been  incorporated  into  the  interface 
utilized  in  the  quiz  system  to  enable  a  quiz  to  administered  following 
an  instructional  lesson.  This  thus  allows  a  quiz -producing  lesson  to 
be  used  as  a  part  of  an  exam  or  a  problem  generator /grader  to  be  used 
as  a  quiz. 


3.   USE  OF  STUDENT  VARIABLES 

All  student  variables  are  available  for  use  by  a  quiz-producing 
lesson  except  nl31  through  nl50;  never  use  student  variables  nl31  through 
nl50. 

A  "quiz  header"  will  occupy  the  first  four  student  variables. 
Portions  of  it  that  are  of  importance  to  the  quiz-producing  lesson  are 
described  on  the  next  page. 


'"Problem  Generator/Grader  for  the  Generative  Exam  System"  contents 
of  which  can  now  be  mostly  found  in  "Documentation  on  the  Generative 
Exam  System",  6/22/76. 
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nl 


.quiz   ////e  ~ 

Quit 

IIIq^IP 

(20 iz.  Score, 

hU.  l>oSS/d/e 

<M»z  4*cS  i/ze 

$(uc/ef)(s  size. 

NOTE:   Never  use  the  shaded  portions.' 


Quiz  title  :   20  character  (or  less)  title  of  the  quiz 
2 


Quiz  number 

2 
Quiz  flag 


Quiz  score 


:   supplied  for  display  on  quiz  pages 

=0  when  a  student  is  first  attempting  the  quiz; 
^0  after  student  exits  quiz,  then  returns  to  make 
changes  to  some  answers 

score  student  earned  on  quiz  (assigned  by  the  quiz- 
producing  lesson  when  student  ends  Task  1) 


Pts.  possible:   maximum  number  of  points  possible  on  the  quiz 

(assigned  by  quiz-producing  lesson  during  Task  k) 

Quiz  specs  size:   number  of  student  variables  needed  by  the  quiz- 
producing  lesson  to  specify  precisely  what  quiz 
questions  to  administer  plus  k   (assigned  by  quiz- 
producing  lesson  during  Task  h) 

Student's  work  size:   number  of  student  variables  needed  to  hold  a 

student's  precise  answers  plus  k   (assigned 
by  the  quiz-producing  lesson  during  Task  k) 


"read  only  -  never  change  its  contents 
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Define  the  first  four  student  variables  as  follows 


define     qh_ttl=nl 

segment, qhl=n3, 5 
qh_num=qhl(l) 
qh_flg=qhl(2) 
segment, qh2=n3,l° 
qh_sco=qh2(3) 
qh_max=qh2(i+) 
qh_spc=qh2(5) 
qh  sws=qh2(6) 


$$quiz  title  (2  words) 

$lj>quiz  number 
$$quiz  flag 

$$score  on  quiz 
$$total  possible  points 
$$quiz  specs  size 
(^student's  work  size 


k.      FORMAT  OF  QUIZ  PAGES 

The  top  of  each  quiz  page  should  appear  similar  to  the 
following  example : 


quiz  number  from 
quiz  header  — > 


1  Question  7  of  9 


quiz  title  from 
quiz  header,*" — v 

fortfmt2  quiz  4/ 


2  points 


At  the  bottom  of  each  page  should  be  information  stating  what 
a  student  can  do  next.  For  example: 


Press:   -NEXT-  for  the  next  question 

-BACK-  for  the  previous  question 
-SHIFT  DATA-  to  quit 
-HELP-  for  some  aid 
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5.   KEY  CONVENTIONS 

To  maintain  uniformity  among  all  quizzes,  the  following 
key  conventions  should  be  adhered  to  as  closely  as  possible: 

For  tasks  1  &  2: 


-SHIFT  DATA-' 
•SHIFT  HELP- 
-HELP- 


-NEXT- 


-BACK- 


3 


-jumpout-  student  to  quiz  system 
present  explanation  about  whole  quiz 
present  explanation  of  current  question 
advance  student  to  next  quiz  page 
take  student  to  previous  quiz  page 


For  task  3: 


An  appropriate  subset  of  the  keys  of  tasks  1  &  2, 


For  task  k: 


■SHIFT  DATA- 
■  SHIFT  LAB- 
■LAB- 


-jumpout-  user  to  the  quiz  system 
review  corrected  quiz  as  a  student 
administer  quiz  as  to  a  student 


6.   RECIPES  FOR  THE  FOUR  TASKS 

The  processing  necessary  in  each  of  the  four  tasks  that 
a  quiz-producing  lesson  must  perform  is  presented  below  in  a  step- 
wise manner.   Minor  variations  may  be  necessitated  depending  upon 
the  particular  quiz. 


These  keys  should  all  accept  a  student  response  for  task  1. 
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Task  1  Processing 

(1)  quiz  flag  should  be  checked  to  determine  whether  or 
not  a  student  is  returning  to  reanswer  questions;  if 
so,  previous  student  responses  follow  the  quiz  header; 
otherwise  quiz  specifications  follow  the  quiz  header 
and  should  be  placed  in  other  student  variables 

(2)  present  question  showing  previous  response 

(3)  receive  new  response 

(k)      save  precise  response  so  showing  previous  response 
(2)  is  possible 

(5)  judge  the  response  (but  don't  tell  student) 

(6 )  if  the  value  of  the  system-defined  variable  -lscore- 
equals  -1,  update  permanent  statistics  to  reflect 
the  student's  answer  (be  sure  only  to  record  the 
student's  last  answer  to  the  question) 

(7)  go  to  2 

The  loop  above  continues  until  student  decides  to  quit. 
Then: 

(8)  quiz  score  should  be  tallied  and  stored  in  quiz  header 

(9)  place  student  responses  in  student  variables  which 
immediately  follow  the  quiz  header  (i.e.  beginning 
with  n5) 

(10)   execute  these  commands:   -inhibit  dropstor- 

-jumpout  return, return - 
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Task  2  Processing 

(1)  [in  case  used  in  exam  system]  quiz  flag  will  be  0  if 
student  did  not  enter  task  1;  otherwise  ^0.   If  0, 
quiz  specs  follow  quiz  header;  otherwise  student 
responses  follow  quiz  header 

(2)  present  question,  student's  response,  and  correct 
response 

(3)  offer  explanations 

(h)      go  to  2 

The  loop  above  continues  until  a  student  decides  to  quit. 

Then: 

(5)   execute  these  commands:   -inhibit  dropstor- 

-jumpout  return, return - 

Task  3  Processing 

(1)  present  accumulated  student  responses  to  each  question 

(2)  when  user  finishes  viewing  data,  execute  the  following 
command:   -jumpout  return, return - 

Task  k   Processing 

(1)  provide  information  about  the  quiz  questions 

(2)  allow  instructor  to  view  and  review  quiz  as  would  a 
student 

(3)  allow  user  to  make  available  quiz  specifications 
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(k)     place  specifications  in  student  variables  immediately- 
following  the  quiz  header  (i.e.  beginning  with  n5) 

(5)  assign  values  to  maximum  points  possible,  quiz  specs 
size,  and  student's  work  size  (all  in  quiz  header) 

(6)  when  user  finishes,  execute:   -jumpout  return, return - 
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APPENDIX  B 

Below  is  a  list  of  the  nine  options  available  to  the  quiz  system 
administrator  through  the  quiz  system  monitor.   Each  option  is 
accompanied  by  a  brief  discussion  of  what  each  involves. 
Option  "l)  Attach  a  quiz  to  a  lesson" 

To  be  chosen  when  a  quiz  is  to  be  "attached"  to 
an  instructional  computer  science  lesson.  User  receives 
access  to  the  pool  of  available  quizzes  from  which  one  may 
be  selected,  designed,  and  "attached".   If  desired,  a 
password  can  be  specified  to  protect  the  quiz  from 
detachment  by  other  system  users.   A  complete  list  of 
changes  that  must  be  made  to  the  lesson  to  enable  proper 
functioning  of  the  lesson/quiz  interface  is  supplied. 
Option  "2)   Detach  a  quiz  from  a  lesson" 

To  be  chosen  when  a  quiz  is  to  be  "detached"  from  an 
instructional  computer  science  lesson.   User  must  supply 
appropriate  password  (if  one  was  specified  earlier  via 
option  l)  in  order  to  deal  the  lesson/quiz  interface  a 
fatal  blow. 
Option  "3)  View  list  of  lessons  with  quizzes  &  attachers" 
To  be  chosen  to  inspect  a  listing  of  all  computer 
science  lessons  currently  utilizing  a  system  quiz.   Lessons 
with  a  quiz  attached  that  for  some  reason  has  been  disabled 
are  identified.   The  identities  of  the  individuals  who 
attached  the  quizzes  are  displayed  as  well  as  the  date 
that  the  attachment  occurred  (Display  12). 
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Lesson  Nam 


--    RTTRCHED   BY    -- 


f    < -COURSE-  ■    on    ■  -DATE 


fort  do 

■•rt  array  2 
ortsubex 
or t char 
or-fc  funct 


anderson 
mont 

anderson 
anderson 
dan i e 1  son 


07.'  05  ■  7t 
06 /l 5/7  6 
06/ 15/7 6 
0  7  ,/0  7  •  7  6 
.6  8  ••- 12  7  6 


indicates  quiz  is  disabled  and  requires  deletior 


end  of  1  is-* 


Display  12:   SAMPLE  LIST  OF  INSTRUCTIONAL 
LESSONS  WITH  QUIZZES 
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Option  "5)  Exit" 

To  be  chosen  to  end  a  visit  to  the  quiz  system  monitor. 

Option  "6)  View  special  system  data" 

To  be  chosen  to  inspect  details  of  the  status  of  the 
quiz  system  to  ensure  its  smooth  operation.   If  detectable 
problems  had  at  some  time  developed,  appropriate  warning 
messages  are  displayed. 

Option  "7)  View  list  of  quiz  lessons  available" 

To  be  chosen  to  inspect  a  listing  of  all  the  lessons 
currently  a  part  of  the  system  that  are  available  to  system 
users  to  produce  quizzes.   Each  lesson  name  is  accompanied  by 
information  specific  to  the  lesson  that  enables  proper 
coordination  between  the  quiz-producing  lesson  and  the 
quiz  system  monitor. 

Option  "8)  Add  a  new  quiz  lesson  (qg/g)" 

To  be  chosen  to  add  a  quiz-producing  lesson  to  the 
pool  of  quizzes  available  in  the  system.  All  information 
necessary  to  incorporate  the  new  lesson  into  the  system 
must  be  entered  (Display  13 ). 

Option  "9)   Delete  a  quiz  lesson  (qg/g)" 

To  be  chosen  to  remove  a  quiz -producing  lesson  from 
the  pool  of  quizzes  available  in  the  system.   An  alert  is 
issued  if  deletion  of  a  specific  quiz -producing  lesson 
would  disable  an  existing  instructional  lesson/quiz  inter- 
face (Display  lU). 


5 


qg/g  is  an  abbreviation  for  "quiz  generator/grader",  a  phrase  which 
technically  describes  the  lesson's  function. 
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Supply  th«2  following  information  about  the  quiz  lesson  (qg/gj 

( 1 )  Lesson  name    fort  f rnt  1  q 

(2)  Twenty  character  description    fort ran  formatting 
Four  entry  points: 

(3)  to  attach  quiz    attch 

(4)  to  give  quiz  £ 

(5)  to   review   quiz         revw 

(6)  to   analyze   quiz  responses        data 


"ype   the   number   of   an    item   to   change    it 

OR 
>ress    -SHIFT   NEXT-    to   add   to   the   system 


Display  13:      SAMPLE  DISPLAY  DURING  ADDITION  OF  A 
QUIZ- PRODUCING  LESSON  TO  THE  SYSTEM 
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Name  the  quiz  lesson  (qg:  g)  you  wish  deleted  from  the  system 

'p     to:  if  Li.  net  q 

rtr^   you.  sure  you  want  to  delete  this  quiz 
lesson  (qg/g) ?   It  appears  you'll  disable  1 
ouiz  (res)  i  f  you  do! 


Press  -SHIFT  HELP-  to  delete 

(The  qu  i  z  (zes)  <  1 1  sab  1  ed  w  i  1  1  be  f  1  agged 
on  the  list  of  lessons  with  quizzes. 

Option  3) 

Press    - BfiCK -    to   ex  i t    w ' t hout    de 1 et  i ng 


Display  lk:      SAMPLE  DISPLAY  DURING  DELETION  OF  A 

QUIZ- PRODUCING  LESSON  FROM  THE  SYSTEM 
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APPENDIX  C 

Certain  facts  about  the  quiz  system's  organization  and 
limitations  remain  to  be  mentioned  to  permit  a  more  complete  view  of 
the  system  and  its  operation.   Since  many  of  these  tidbits  are  quite 
unrelated,  all  are  presented  as  elements  of  a  list: 

-The  maximum  number  of  instructional  lessons  that  can 
be  issued  a  quiz  by  the  system  is  fifty.  However, 
due  to  variations  that  are  possible  in  the  amount  of 
storage  space  required  to  handle  different  instructional 
lesson /quiz  interfaces,  the  actual  number  of  lessons 
that  can  be  accomodated  may  actually  be  less  than  fifty. 

-The  maximum  number  of  quiz-producing  lessons  that  can 
be  made  available  to  system  users  is  fifty. 

-Data  to  indicate  average  quiz  scores,  numbers  of  students 
who  have  taken  a  quiz,  and  average  lengths  of  time 
required  for  students  to  take  a  quiz  is  accumulated 
separately  for  each  course  as  long  as  the  quiz  system's 
storage  space  holds  out.   Since  a  vast  amount  of  space 
is  reserved  to  hold  this  type  of  data,  space  scarcity 
should  be  rare.   Nevertheless,  when  space  limitations 
permit  no  further  new  course  total  accumulations  for 
a  quiz,  all  such  data  is  accumulated  in  general  chunks. 
Data  accumulated  in  this  manner  is  reported  as  such 
whenever  quiz  data  is  displayed. 
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-Detachment  of  a  quiz  from  an  instructional  lesson  or 
deletion  of  a  quiz-producing  lesson  from  the  pool 
of  such  lessons  available  in  the  system  immediately 
frees  all  the  released  system  space  for  other  system 
use.   No  garbage  collection  routines  are  necessary. 

-All  internal  modifications  to  the  system  (e.g.   storing 
gathered  student  data;  saving  information  supplied  by 
a  system  user  to  enable  an  instructional  lesson/quiz 
interface)  are  protected  both  against  abort  (-SHIFT  STOP-) 
attempts  by  students  or  system  users  and  against  simultaneous 
modifications  (e.g.   if  more  than  one  system  user  attempts 
to  attach  a  quiz  to  an  instructional  lesson  at  the  same 
time). 

-Data  made  available  by  the  administration  of  a  quiz  is 
retained  only  the  first  time  a  student  attempts  a  quiz. 
This  is  achieved  by  saving  the  student's  quiz  score  in 
his  or  her  individual  router  variables  enabling  a  check 
to  see  if  a  quiz  score  has  yet  been  recorded.   An 
instructor  of  a  course  thus  has  access  to  individual 
student  quiz  scores,  but  no  additional  score  can  be 
issued  by  the  instructional  lesson  itself. 

-When  a  student  completes  a  quiz  and  control  is  transferred 
back  to  the  instructional  lesson,  all  student  variables  are 
reinstated  exactly  as  they  existed  prior  to  the  keypresses 
that  commenced  quiz  administration.  • 
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-The  quiz  system  in  its  entirety  consists  of  only 
two  programs  (i.e.  disregarding  the  individual  quiz- 
producing  lessons),   "csqdata",  as  mentioned  before, 
comprises  the  quiz  system  monitor  that  is  accessed 
directly  by  both  the  quiz  system  users  and  the  quiz 
system  administrator,   "csxquiz"  is  the  smaller  system 
program  that  handles  the  interface  between  an  instructional 
lesson  and  a  quiz. 
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APPENDIX  D 

The  quizzes  listed  below  either  are  currently  available  in 
the  quiz  system  or  are  currently  being  developed  for  such  availability. 
The  authors  listed  are  those  individuals  responsible  at  least  for  most 
of  the  code  that  produces  the  quizzes;  in  most  cases,  substantial  aid 
and  consultation  in  such  areas  as  question  development  and  quiz  structure 
was  provided  by  either/both  this  author  or /and  R.  G.  Montanelli,  Jr. 


Quiz  Topic 
FORTRAN  character  manipulation 
FORTRAN  2-dimensional  arrays 
FORTRAN  DO  loops 
FORTRAN  subroutines 
FORTRAN  arithmetic 
FORTRAN  functions 
FORTRAN  formatting  (elementary) 


Quiz  Author 

R.  I.  Anderson 

D.  Gamage 

J.  Miller 

T.  Puterbaugh 

J.  Miller 

J.  Glauert 

R.  Bogue 


The  above  list  could  possibly  be  expanded  due  to  the 
existance  of  L.  R.  Whitlock's  generative  exam  system.   Details  of 
the  interface  that  enables  a  quiz's  administration  following  an 
instructional  computer  science  lesson  were  derived  from  details  of 
the  interface  that  enables  the  administration  of  an  exam  in  the 
PLATO  generative  exam  system.   Certain  aspects  of  both  interfaces 
were  implemented  sufficiently  identical  such  that  a  problem  set 


L.  R.  Whitlock,  "interactive  Test  Construction  and  Administration  in  the 
Generative  Exam  System",  University  of  Illinois,  Department  of  Computer 
Science,  Report  No.  UIUCDCS-R-76-821. 
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available  for  use  as  a  portion  of  an  exam  in  the  exam  system  could 
actually  be  made  available  for  use  as  a  quiz  in  the  quiz  system  (and 
vice  versa).   Since  all  problem-set-producing  lessons  do  not  incorporate 
question  response  gathering  facilities  as  do  the  quiz -producing  lessons, 
alterations  to  the  exam  system  lessons  -would  be  necessary  prior  to  such 
an  exchange.   The  full  potential  of  these  exchanges  is  yet  to  be 
investigated. 


Whitlock  refers  to  these  as  problem  generator/graders. 
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